Analysis of uploads from Cholera JHU-Lee for touchstone 201910gavi

Objective: To assist the evaluation of central estimates produced by modellers, following the full model runs in Oct 2019, according to the VIMC minimum standards.

Checks:

Click to show/hide all passed checks

The following automated checks passed:
2 cases for cholera-no-vaccination-test is greater than for cholera-campaign-default-test (see Table 7)
4 cases for cholera-no-vaccination-test is greater than for cholera-campaign-default-test (see Table 7)
12 deaths for cholera-no-vaccination-test is greater than for cholera-campaign-default-test (see Table 7)
11 deaths for cholera-no-vaccination-test is greater than for cholera-campaign-default-test (see Table 7)
13 dalys for cholera-no-vaccination-test is greater than for cholera-campaign-default-test (see Table 7)
111 dalys for cholera-no-vaccination-test is greater than for cholera-campaign-default-test (see Table 7)

The Model

Specific characteristics of this model are shown in Table 1.

  details
disease_name Cholera
modelling_group JHU-Lee
model_name JHU-Lee-Cholera
model_type static
Vaccine_1 Cholera (SIA)
Vaccine_2 NA
Vaccine_3 NA
burden_min_age 0
burden_max_age 100
cohort_first 1900
cohort_last 2100
year_min 2000
year_max 2100
gender_specific No
gender Both
number_countries 5
number_expected_outcomes 4
1 cases
2 dalys
3 deaths
4 cohort size
number_expected_scenarios 2
21 Campaign, Default
Table 1: General characteristics of the model.

Scenarios

The modellers have been given 2 vaccination scenarios for this disease (Cholera). Each scenario contains one or more coverage sets. The characteristics of the scenarios are explained in the table below.

scenario coverage_set vaccine activity_type
Campaign, Default Cholera: Cholera, with, campaign; test Cholera campaign
No vaccination No vaccination NA NA
Table 2: Scenario description and coverage sets.

Coverage Sets

The trends of the specific coverage sets can be observed in Fig 1. Notice that for campaign scenarios the coverage can be more than 100% in some cases for the interpolated population for that country/year/age combination.

Age ranges of the different coverage sets include either standard age ranges (as it is the case of Measles) but sometimes it differs by year and country so that several combinations can be possible (See table below). This is key to calculate the target population of a specific vaccine for an age/year/country combination.

coverage_set_name age_from age_to freq
Cholera: Cholera, with, campaign; test 1 100 104
Table 3: Coverage sets and age ranges.

Fully vaccinated persons (FVPs)

The estimates of fully vaccinated persons (FVPs) are obtained by multiplying coverage from each coverage set and the target population.

 FVPs=coverage∗target population
where target population for routine activities is interpolated UNWPP population (from Montagu: modellers contribution portal) and for campaign activities, it is specified in the coverage files for campaign.

This calculation should reflect the different coverage sets included in each scenario. For details on the FVPs in all coverage scenarios by country, see Supplementary Figure 1.

Outcomes

The modellers have been given 3 outcomes for (Cholera). The characteristics of these outcomes are explained in the table below. Please notice that for some models the outcomes comprise several sub-outcomes that are reported in a separate form.

outcome definition
cases Cholera cases
dalys Cholera dalys
deaths Cholera deaths

Table 4: Outcome definitions.

Part A. Checking general upload

This section is based on the uploaded scenarios where we also check that the estimates uploaded correspond to the expected outcome.

Table 5 shows the scenarios uploaded (specified in Table 2) and the specific touchstone for tracking purposes.

scenario_description touchstone touchstone_name uploaded_by
No vaccination 201910gavi-5 201910gavi elizabeth.lee
Campaign, Default 201910gavi-5 201910gavi elizabeth.lee
Table 5: Technical information for tracking purposes.

Age and year ranges and countries uploaded.

Age and year ranges and the number of countries uploaded should correspond to the numbers expected given in Table 1.

min_age_upload 0
max_age_upload 100
min_year_upload 2000
max_year_upload 2100
num_countries 5

Expected number of rows

scenario_description outcome_code number of rows
No vaccination cases 51005
No vaccination cohort_size 51005
No vaccination dalys 51005
No vaccination deaths 51005
Campaign, Default cases 51005
Campaign, Default cohort_size 51005
Campaign, Default dalys 51005
Campaign, Default deaths 51005
Table 6: Summary of central estimates uploaded by scenario and outcome.

The expected 'number of rows' in this table is 51005, (age classes x years x countries). If the number expected is different from the uploaded, please compare Table 1 against Table 3, and take a look at Fig. 1, 2 and 3 for some insight.

In Fig.3, we are comparing the observations expected for this model (blue line) against what has been uploaded by the modeller (orange area/dots). The missing/excess observations correspond to the area between the blue and orange lines. The figure aims to represent the birth cohort being followed over time, so that in a dynamic model we are expecting a rectangular shape whereas for a static model a 'trapezoidal' shape is the minimum expected. Some static models can also have a rectangular shape.

In Fig. 4 we are checking the patterns of zero values in the upload.

Zero values are normal in many cases, and we expect either absence, minimum presence or a random pattern of them. When a non-random pattern of zero values is observed, this plot can help in interpreting some of the intentional assumptions or systematic errors that cannot be seen in Fig. 3 (i.e. absence of estimates of an outcome in some age classes or in a whole birth cohort).

Part B. Checking cohort size

The cohort size is the number of people alive in a given birth cohort specified by the calendar year and age during that year, so it will be the same across all scenarios. We will then be able to calculate the number of FVPs (fully vaccinated persons) by multiplying this cohort size with the relevant coverage. The cohort size should be comparable to the interpolated population provided on Montagu. The cohort size should reflect the age range, time range and gender (female, male or both) for which the model is tracking the population.

In this section, the plots are specific to each model in terms of age range, year range, countries and gender. For each plot, the left panel reflects the demography figures provided in Montagu (interpolated population), the central panel represents the cohort size uploaded by modellers, and the right panel represents the difference between the two, given by cohort size - interpolated population. If the difference is positive, then the model has more cohorts than the interpolated population, and if negative, the model has fewer cohorts compared to the interpolated population.

The population used here from Montagu is type int_pop with source dds-201910 between 2000 and 2100.

Comparison of cohort size against interpolated population in all countries

Comparison of cohort size against interpolated population in PINE countries

Fig. 7 and 8 show absolute and relative difference respectively, between interpolated population and cohort size by country for this model.

Part C. Checking patterns of the specific estimates

In this section we are checking burden estimates for the different scenarios.

Age patterns

Here, we look at the aggregated (for all countries) age distribution patterns by outcome.

Global burden values by decades 2000-2030

Figure 10 shows the total estimates of burden by decades between 2000-2030 (for the total number of countries) to see if the estimates are reasonable for the different scenarios.

.

Global burden disease values 2000-2100

Total burden values

Table 7 shows the total burden from all countries and age groups for no vaccination and vaccination scenarios. These values should correspond to the total across all ages from Fig. 11.

outcome_code scenario_description number_of_people_millions
cases No vaccination 17.28
cases Campaign, Default 16.65
cohort_size No vaccination 54384
cohort_size Campaign, Default 54384
dalys No vaccination 25.94
dalys Campaign, Default 25.13
deaths No vaccination 0.6034
deaths Campaign, Default 0.5808
Table 7: Total burden for each scenario across all countries and ages.

FVPs 2000-2100 (PINE countries)

Table 8 shows the total fvps for each scenario in the PINE countries for the same period as shown in Fig. 12. These values should correspond to the total across all years from Fig. 12.

scenario_description country fvp_in_millions
cholera-campaign-default-test ETH 28.88
cholera-campaign-default-test IND 1180
cholera-campaign-default-test NGA 47.1
Table 8: Total FVPs estimates for each scenario for PINE countries in millions.

Disease burden (PINE countries)

Disease burden values (PINE countries)

Table 9 shows total burden estimates for all years and ages for each scenario from PINE (Pak-Pakistan, IND-India, NGA-Nigeria and ETH-Ethiopia) countries.The values should correspond to the total values from Fig. 13 for each scenario and burden outcome.

country scenario_description burden_outcome thousands
ETH No vaccination cases 1321
ETH Campaign, Default cases 1306
ETH No vaccination dalys 842.6
ETH Campaign, Default dalys 834.3
ETH No vaccination deaths 16.48
ETH Campaign, Default deaths 16.32
Table 9: Total burden for estimates of each scenario for PINE countries in thousands.

No vaccination (fixed scale)

The figure below shows burden for each outcome from the No vaccination scenario for all countries.

Supplementary Information

Country checking

Countries expected vs uploaded
ISO3 Country cholera-campaign-default-test cholera-no-vaccination-test
COD Congo, the Democratic Republic of the 1 / 1 1 / 1
ETH Ethiopia 1 / 1 1 / 1
KEN Kenya 1 / 1 1 / 1
SOM Somalia 1 / 1 1 / 1
SSD South Sudan 1 / 1 1 / 1
TOTAL ALL COUNTRIES 5 / 5 5 / 5

Fully Vaccinated Persons (FVPs) by country